{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/peremartra/Large-Language-Model-Notebooks-Course/blob/main/6-PRUNING/6_5b_pruning_depth_st_llama3.2-1b-Instruct_OK.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DT0FAsUnncFY"
      },
      "source": [
        "<div>\n",
        "    <h1>Large Language Models Projects</a></h1>\n",
        "    <h3>Apply and Implement Strategies for Large Language Models</h3>\n",
        "    <h2>Pruning Llama 3.2. (Adapted to Instruct Models)</h2>\n",
        "    <h3>Example of approach to depth pruning a Llama Model.</h3>\n",
        "</div>\n",
        "\n",
        "by [Pere Martra](https://www.linkedin.com/in/pere-martra/)\n",
        "\n",
        "_______\n",
        "Models: meta-llama/Llama-3.2-1B\n",
        "\n",
        "Colab Environment: GPU T4.\n",
        "\n",
        "Keys:\n",
        "* Pruning\n",
        "* Structured pruning\n",
        "* Depth pruning.\n",
        "\n",
        "\n",
        "Related article:\n",
        "_______\n",
        "**disclaimer: The pruning section was created after the first edition of the book was published. They are not included in the book’s original content but are intended to supplement and expand on the topics covered.**\n",
        "\n",
        "This is the unofficial repository for the book:\n",
        "        <a href=\"https://amzn.to/4eanT1g\"> <b>Large Language Models:</b> Apply and Implement Strategies for Large Language Models</a> (Apress).\n",
        "        The book is based in this repository, but the notebooks are being updated, and I am incorporating new examples and chapters.\n",
        "        If you are looking for the official repository for the book, with the original notebooks, you should visit the\n",
        "        <a href=\"https://github.com/Apress/Large-Language-Models-Projects\">Apress repository</a>, where you can find all the notebooks in their original format as they appear in the book.\n",
        "______"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iEwxZCVsoIau"
      },
      "source": [
        "# Introduction\n",
        "This notebook cotinues the work done at: [6_3_pruning_structured_llama3.2-1b_OK.ipynb](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/6-PRUNING/6_3_pruning_structured_llama3.2-1b_OK.ipynb) a width pruning was applied to a Llama3.2 model.\n",
        "\n",
        "In this notebook, we will look at an example of depth pruning, which involves removing entire layers from the model.\n",
        "\n",
        "The first thing to note is that removing entire layers from a transformer model usually has a significant impact on the model's performance. This is a much more drastic architectural change compared to the simple removal of neurons from the MLP layers, as seen in the previous example.\n",
        "\n",
        "For this reason, these models are not designed to be used directly after the pruning process. Instead, they will require a subsequent fine-tuning process to recover their capabilities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eQIxAOPZtPBN"
      },
      "source": [
        "#Install libraries & Configure variables."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5zHApVm41HWq"
      },
      "outputs": [],
      "source": [
        "!pip install -q transformers\n",
        "!pip install -q torch\n",
        "!pip install -q datasets\n",
        "!pip install -q sentencepiece  # Required for LLaMA tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GJNgRj4M187E"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from datasets import load_dataset\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "from torch import nn\n",
        "from torch.utils.data import DataLoader\n",
        "import os\n",
        "from tqdm import tqdm"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tbIyUlXEtbqs",
        "outputId": "f984c3f6-9cd5-4766-8f2a-a550835e617d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using device: cuda\n"
          ]
        }
      ],
      "source": [
        "# Check if GPU is available\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "print(f\"Using device: {device}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sM-QwxyKw-YG"
      },
      "source": [
        "#Download model and explore structure"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q-z_1Zpg2I6u"
      },
      "outputs": [],
      "source": [
        "model_name = 'meta-llama/Llama-3.2-1B-Instruct'\n",
        "model = AutoModelForCausalLM.from_pretrained(model_name,\n",
        "                                             dtype=torch.float16).to(device)\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "tokenizer.pad_token = tokenizer.eos_token  # Set pad token"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9UpMD4Hw2MWg"
      },
      "outputs": [],
      "source": [
        "def get_output(prompt, model=model, tokenizer=tokenizer):\n",
        "    # Chat forma for modelInstruct\n",
        "    messages = [\n",
        "        {\"role\": \"user\", \"content\": prompt}\n",
        "    ]\n",
        "\n",
        "    # Aply chat template\n",
        "    inputs = tokenizer.apply_chat_template(\n",
        "        messages,\n",
        "        add_generation_prompt=True,\n",
        "        return_tensors='pt'\n",
        "    ).to(device)\n",
        "\n",
        "    outputs = model.generate(\n",
        "        inputs,\n",
        "        max_length=150,\n",
        "        num_return_sequences=1,\n",
        "        pad_token_id=tokenizer.pad_token_id,\n",
        "        temperature=None,\n",
        "        top_p=None,\n",
        "        do_sample=False,\n",
        "        num_beams=5,\n",
        "        early_stopping=True,\n",
        "        no_repeat_ngram_size=2\n",
        "    )\n",
        "    generated_tokens = outputs[0][len(inputs[0]):]\n",
        "    generated = tokenizer.decode(generated_tokens, skip_special_tokens=True)\n",
        "\n",
        "\n",
        "    return generated.strip()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4muyx_8M5OAu"
      },
      "source": [
        "## studying the model structure\n",
        "As demonstrated in the [previous notebook](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/6_2_pruning_structured_llama3.2-1b_KO.ipynb), studying the structure of the model that will undergo pruning is crucial.\n",
        "\n",
        "In this notebook, we’re going to fine-tune the pruning process for the Llama3.2 model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y5Hs4oQ4B7Z0",
        "outputId": "9037c520-9421-4345-d568-fc0d0b8c505a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LlamaForCausalLM(\n",
            "  (model): LlamaModel(\n",
            "    (embed_tokens): Embedding(128256, 2048)\n",
            "    (layers): ModuleList(\n",
            "      (0-15): 16 x LlamaDecoderLayer(\n",
            "        (self_attn): LlamaAttention(\n",
            "          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
            "          (k_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
            "          (v_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
            "          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
            "        )\n",
            "        (mlp): LlamaMLP(\n",
            "          (gate_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
            "          (up_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
            "          (down_proj): Linear(in_features=8192, out_features=2048, bias=False)\n",
            "          (act_fn): SiLUActivation()\n",
            "        )\n",
            "        (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "        (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "      )\n",
            "    )\n",
            "    (norm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "    (rotary_emb): LlamaRotaryEmbedding()\n",
            "  )\n",
            "  (lm_head): Linear(in_features=2048, out_features=128256, bias=False)\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yPMslK3QCAb1"
      },
      "source": [
        "\n",
        "Each layer of the model consists of the attention section and the MLP section. When we remove a layer, the entire layer is eliminated. This is a crucial point because finding a balance in selecting which layers to remove will be challenging.\n",
        "\n",
        "Later, in future notebooks, you will learn about different techniques for selecting layers, based on their activation and how the model responds to a specific dataset.\n",
        "\n",
        "In this notebook, you will explore three different layer selection techniques:\n",
        "\n",
        "- Summing the magnitudes of all the weights in the layers.\n",
        "- Removing the first layers of the model.\n",
        "- Removing the last layers of the model.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "alKH3QH64WFL",
        "outputId": "32e35809-d9e8-4a16-f53c-e3158c41e4c2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Generated text: Paris is both the largest city in France and its capital, as it is where the French government is located.\n"
          ]
        }
      ],
      "source": [
        "# Test the original model\n",
        "prompt = \"What is the capital of France?\"\n",
        "generated = get_output(prompt)\n",
        "print(f\"Generated text: {generated}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Generated text: Paris is both the largest city in France and its capital, as it is where the French government is located.\n",
        "\n"
      ],
      "metadata": {
        "id": "PfQuOoSIIa_4"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8WR96iwq2XYH"
      },
      "outputs": [],
      "source": [
        "def count_parameters(model):\n",
        "    return sum(p.numel() for p in model.parameters())\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kph43oObnet7",
        "outputId": "fb6348d7-8d28-4127-dd9a-2e37ba5aec92"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original model parameters: 1235814400\n"
          ]
        }
      ],
      "source": [
        "original_param_count = count_parameters(model)\n",
        "print(f\"Original model parameters: {original_param_count}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CK9NwmBWnkSP"
      },
      "source": [
        "#Pruning the Model.\n",
        "##Support pruning functions.\n",
        "\n",
        "Here are three differeten methods I used to calculate wich layers to mantain.\n",
        "\n",
        "We remove the last layers of the models preserving the last one.\n",
        "\n",
        "In a Transformer model like this, layers are organized hierarchically: the initial layers (closer to the input) tend to capture basic language patterns (such as syntactic structure, common word combinations, etc.), while the intermediate and final layers refine these representations, capturing higher-order relationships, global coherence, and subtle semantic nuances.\n",
        "\n",
        "Removing the initial layers directly undermines the foundation upon which more complex representations are built, leading the model to generate meaningless text sequences. Similarly, removing layers based on weight importance metrics (without considering their position or function) can eliminate layers critical for linguistic cohesion or contextual coherence.\n",
        "\n",
        "On the other hand, removing the final layers, while resulting in a loss of some refinement and specialization capabilities, preserves the initial and middle layers that have already learned fundamental language rules and basic word dependencies.\n",
        "\n",
        "However, this is just an empirical and highly simple test. Later, when we evaluate the model's performance using rankings, we will see that it retains a significant portion of its characteristics. Therefore, we are dealing with a model that can deliver very good results after a small fine-tuning process to recover some of the lost capabilities."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NX9Boph94RWA"
      },
      "outputs": [],
      "source": [
        "# ELIMINATE LAST LAYERS OF THE MODEL.\n",
        "def prune_last_layers(model, num_layers_to_remove):\n",
        "    \"\"\"\n",
        "    Removes the last 'num_layers_to_remove' layers from the model.\n",
        "\n",
        "    Args:\n",
        "    - model: The model from which layers will be pruned.\n",
        "    - num_layers_to_remove: Number of layers to remove from the top of the stack.\n",
        "\n",
        "    Returns:\n",
        "    - model: The pruned model with fewer layers.\n",
        "    \"\"\"\n",
        "    total_layers = len(model.model.layers)\n",
        "\n",
        "    # Ensure we are not removing more layers than exist\n",
        "    if num_layers_to_remove >= total_layers:\n",
        "        raise ValueError(\"Number of layers to remove is greater or equal to total layers.\")\n",
        "\n",
        "    # Slice the layers to remove the last ones\n",
        "    new_layers = model.model.layers[:total_layers - num_layers_to_remove]\n",
        "    model.model.layers = nn.ModuleList(new_layers)\n",
        "\n",
        "    # Update the model configuration\n",
        "    model.config.num_hidden_layers = len(model.model.layers)\n",
        "\n",
        "    return model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QT0v_RpeST87"
      },
      "source": [
        "# Prune Loop\n",
        "The update_model function iterates through the blocks within the model's Transformer structure. This structure consists of multiple `LlamaDecoderLayer` blocks, and each of these blocks contains a pair of `LlamaSdpaAttention` and `LlamaMLP` components.\n",
        "```\n",
        "(layers): ModuleList(\n",
        "      (0-15): 16 x LlamaDecoderLayer(\n",
        "        (self_attn): LlamaSdpaAttention(\n",
        "          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
        "          (k_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
        "          (v_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
        "          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
        "          (rotary_emb): LlamaRotaryEmbedding()\n",
        "        )\n",
        "        (mlp): LlamaMLP(\n",
        "          (gate_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
        "          (up_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
        "          (down_proj): Linear(in_features=8192, out_features=2048, bias=False)\n",
        "          (act_fn): SiLU()\n",
        "        )\n",
        "        (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
        "        (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
        "      )\n",
        "  )    \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FxJEWg1X3j0m"
      },
      "outputs": [],
      "source": [
        "def update_model(model, prune_percent):\n",
        "    \"\"\"\n",
        "    Modifies the model by removing entire layers from the end of the stack,\n",
        "    instead of basing it on importance scores. This is a heuristic approach\n",
        "    that tries pruning the top layers to see if it yields better results.\n",
        "\n",
        "    Args:\n",
        "    - model: Model to prune.\n",
        "    - prune_percent: Percentage of layers to prune from the top.\n",
        "\n",
        "    Returns:\n",
        "    - model: New pruned model with fewer layers.\n",
        "    \"\"\"\n",
        "    ### uncomment this if you want to use weight layer selection.\n",
        "    #model = prune_layers(model, prune_percent)\n",
        "\n",
        "    total_layers = len(model.model.layers)\n",
        "    num_layers_to_remove = int(total_layers * prune_percent)\n",
        "\n",
        "    model = prune_last_layers(model, num_layers_to_remove)\n",
        "    #model = prune_first_layers(model, num_layers_to_remove)\n",
        "\n",
        "    # Update the model configuration to reflect the new number of layers\n",
        "    model.config.num_hidden_layers = len(model.model.layers)\n",
        "    return model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KtHtSbRmS267"
      },
      "source": [
        "## Obtain & test the pruned model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIUnFU5R3n42",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "20e6e61c-1013-4af9-9e3d-6772cd9da8fe"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LlamaForCausalLM(\n",
              "  (model): LlamaModel(\n",
              "    (embed_tokens): Embedding(128256, 2048)\n",
              "    (layers): ModuleList(\n",
              "      (0-12): 13 x LlamaDecoderLayer(\n",
              "        (self_attn): LlamaAttention(\n",
              "          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
              "          (k_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
              "          (v_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
              "          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
              "        )\n",
              "        (mlp): LlamaMLP(\n",
              "          (gate_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
              "          (up_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
              "          (down_proj): Linear(in_features=8192, out_features=2048, bias=False)\n",
              "          (act_fn): SiLUActivation()\n",
              "        )\n",
              "        (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
              "        (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
              "      )\n",
              "    )\n",
              "    (norm): LlamaRMSNorm((2048,), eps=1e-05)\n",
              "    (rotary_emb): LlamaRotaryEmbedding()\n",
              "  )\n",
              "  (lm_head): Linear(in_features=2048, out_features=128256, bias=False)\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 63
        }
      ],
      "source": [
        "import copy\n",
        "prune_percent = 0.2  # Prune 20% of neurons\n",
        "pruned_model = update_model(copy.deepcopy(model), prune_percent)\n",
        "pruned_model.eval()\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "temp_dir = \"./temp_pruned_model\"\n",
        "pruned_model.save_pretrained(temp_dir)\n",
        "tokenizer.save_pretrained(temp_dir)\n",
        "del pruned_model\n",
        "torch.cuda.empty_cache()\n",
        "pruned_model = AutoModelForCausalLM.from_pretrained(\n",
        "    temp_dir,\n",
        "    dtype=torch.float16, # Make sure to match original load settings\n",
        "    device_map=device\n",
        ")"
      ],
      "metadata": {
        "id": "gE52bQNnLiKf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tdJUkfWI3qMM",
        "outputId": "642e7629-0c7d-4ed0-bfb2-960a41f404b5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pruned model parameters: 1053349888\n",
            "Reduction in parameters: 182464512\n",
            "Percentage of weight savings: 14.76%\n"
          ]
        }
      ],
      "source": [
        "# Recalculate the number of parameters\n",
        "pruned_param_count = count_parameters(pruned_model)\n",
        "reduction_in_params = original_param_count - pruned_param_count\n",
        "percentage_savings = (reduction_in_params / original_param_count) * 100\n",
        "\n",
        "print(f\"Pruned model parameters: {pruned_param_count}\")\n",
        "print(f\"Reduction in parameters: {reduction_in_params}\")\n",
        "print(f\"Percentage of weight savings: {percentage_savings:.2f}%\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wvj-iIsO5M6U",
        "outputId": "b72213d9-82b1-4b04-8d54-9e5a401e064a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Generated text after pruning: Paris is indeed considered as capital city of france however technically speaking capitals capitals are usually referred to as Capitals capitals themselves rather than just Capitals themselves because typically there are multiple capitals depending upon how many countries are involved amongst others such as Vatican City which is seperate from mainland europe however amongst european nationssuch as france france itself serves as both capitals whilst others like france also recogniseother countries like France as their own respective capitals additionally sometimes termed as administrative capitals or administrative centers depending on context however generally speaking amongst internationally recognised sovereign nations like countries internationally recognized as nations themselvessuch\n"
          ]
        }
      ],
      "source": [
        "# Test the pruned model\n",
        "generated = get_output(prompt, pruned_model, tokenizer)\n",
        "print(f\"Generated text after pruning: {generated}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Generated text after pruning: Paris is indeed considered as capital city of france however technically speaking capitals capitals are usually referred to as Capitals capitals themselves rather than just Capitals themselves because typically there are multiple capitals depending upon how many countries are involved amongst others such as Vatican City which is seperate from mainland europe however amongst european nationssuch as france france itself serves as both capitals whilst others like france also recogniseother countries like France as their own respective capitals additionally sometimes termed as administrative capitals or administrative centers depending on context however generally speaking amongst internationally recognised sovereign nations like countries internationally recognized as nations themselvessuch\n"
      ],
      "metadata": {
        "id": "rL7BLl7dHZ4U"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JGzXMQrVTULv"
      },
      "source": [
        "The result is realy different from what the original model produced, and is far to be a fairly accurate response, but at least is understable text.\n",
        "\n",
        "In contrast to the model created in notebook: [6_2_pruning_structured_llama3.2-1b_KO.ipynb](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/6_2_pruning_structured_llama3.2-1b_KO.ipynb) where the pruned Llama model lost almost all its utility, the model in this notebook retains a good portion of its knowledge."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dDQrSrf-VCyI"
      },
      "source": [
        "Looking at the model’s new structure, we can see that now the model have only 13 layers instead of the 16 original. So we removed 3 entire layers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ATAiqZW30NYN",
        "outputId": "e62d0619-2ab5-4e1f-c440-3d6de9671558"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LlamaForCausalLM(\n",
            "  (model): LlamaModel(\n",
            "    (embed_tokens): Embedding(128256, 2048)\n",
            "    (layers): ModuleList(\n",
            "      (0-12): 13 x LlamaDecoderLayer(\n",
            "        (self_attn): LlamaAttention(\n",
            "          (q_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
            "          (k_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
            "          (v_proj): Linear(in_features=2048, out_features=512, bias=False)\n",
            "          (o_proj): Linear(in_features=2048, out_features=2048, bias=False)\n",
            "        )\n",
            "        (mlp): LlamaMLP(\n",
            "          (gate_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
            "          (up_proj): Linear(in_features=2048, out_features=8192, bias=False)\n",
            "          (down_proj): Linear(in_features=8192, out_features=2048, bias=False)\n",
            "          (act_fn): SiLUActivation()\n",
            "        )\n",
            "        (input_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "        (post_attention_layernorm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "      )\n",
            "    )\n",
            "    (norm): LlamaRMSNorm((2048,), eps=1e-05)\n",
            "    (rotary_emb): LlamaRotaryEmbedding()\n",
            "  )\n",
            "  (lm_head): Linear(in_features=2048, out_features=128256, bias=False)\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "print(pruned_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q6qEmvooZycx"
      },
      "source": [
        "#Upload the model to HuggingFace."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S2Ll_kqe5QzO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ead33b23-1bae-4317-fba4-7830dd591697"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pruned model saved to ./depth20-llama-3.2-1b-Instruct\n"
          ]
        }
      ],
      "source": [
        "new_model_name = 'depth20-llama-3.2-1b-Instruct'\n",
        "output_dir = './'+new_model_name\n",
        "if not os.path.exists(output_dir):\n",
        "    os.makedirs(output_dir)\n",
        "\n",
        "pruned_model.save_pretrained(output_dir)\n",
        "tokenizer.save_pretrained(output_dir)\n",
        "print(f\"Pruned model saved to {output_dir}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3LjjsGZV5ZHJ"
      },
      "outputs": [],
      "source": [
        "# Push the model to your Hugging Face repository\n",
        "\n",
        "pruned_model.push_to_hub(new_model_name, private=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6xNN-aYa5h9B"
      },
      "outputs": [],
      "source": [
        "tokenizer.push_to_hub(new_model_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XdKFR5Ju23kI"
      },
      "source": [
        "#Evaluating models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_UM2pkqFAYEe"
      },
      "source": [
        "In this section, we'll take a look at some standard evaluations in the world of Large Language Models using the lm-evaluation library from EleutherAI.\n",
        "\n",
        "Specifically, we'll use LAMBADA, ARC_EASY and BoolQ. Since the pruning performed could be considered structural—that is, it affects the model's overall structure without a specific target—I’ve chosen rather different evaluation tasks.\n",
        "\n",
        "I want to remind you that the goal of this notebook is to demonstrate the pruning process, so I won’t be doing a comprehensive study of how it impacts performance; that will be saved for a future article. Additionally, these models are designed to be fine-tuned before being used.\n",
        "\n",
        "However, I believe that seeing how pruning impacts model performance can help illustrate the pruning process itself.\n",
        "\n",
        "The model selected for the comparision is the one with a 20% pruning using the last layers selection."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XPOg8Hoa22xA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3e74c60e-812f-48ed-ac3b-af78906dad45"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.6/53.6 kB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m51.8/51.8 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.5/7.5 MB\u001b[0m \u001b[31m83.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m491.5/491.5 kB\u001b[0m \u001b[31m44.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.1/84.1 kB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m293.6/293.6 kB\u001b[0m \u001b[31m30.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m104.1/104.1 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m91.1/91.1 kB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for rouge-score (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for word2number (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "!pip install -q lm-eval\n",
        "from lm_eval import evaluator, tasks, models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "N5wp8lM63IGz"
      },
      "outputs": [],
      "source": [
        "def evaluate_hf_model(model_name, tasks=['arc_easy'], num_fewshot=0):\n",
        "    \"\"\"\n",
        "    It calls the evaluator to evaluate a model available on Hugging Face.\n",
        "\n",
        "    Args:\n",
        "    - model_name: The model name in hugging Face.\n",
        "    - tasks: Tasks to evaluate.\n",
        "    - num_fewshot: Number of examples of few-shot learning\n",
        "\n",
        "    Returns:\n",
        "    - metrics.\n",
        "    \"\"\"\n",
        "    model_args = f\"pretrained={model_name},device=cuda\"\n",
        "    tasks = tasks\n",
        "\n",
        "    results = evaluator.simple_evaluate(\n",
        "      model=\"hf\",\n",
        "      model_args=model_args,\n",
        "      tasks=tasks,\n",
        "      num_fewshot=0,  # Number of few-shot smaples.\n",
        "      limit=None,  # Use all the samples in the Evaluate Dataset.\n",
        "      bootstrap_iters=10\n",
        "    )\n",
        "\n",
        "    metrics = results.get('results', {})\n",
        "    return metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yZm8VvA33Nh6"
      },
      "outputs": [],
      "source": [
        "# Select tasks to evaluate.\n",
        "tasks = ['lambada', 'boolq', 'arc_easy']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kTW3E3mp145e"
      },
      "outputs": [],
      "source": [
        "metrics_pruned = evaluate_hf_model(\"oopere/depth20-llama-3.2-1b-Instruct\", tasks=tasks)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w-g3vyPN3VZp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2e05fefb-33d7-4f06-8b56-52cc1998d969"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'arc_easy': {'alias': 'arc_easy',\n",
              "  'acc,none': 0.48737373737373735,\n",
              "  'acc_stderr,none': 0.010256511718330434,\n",
              "  'acc_norm,none': 0.47474747474747475,\n",
              "  'acc_norm_stderr,none': 0.010246690042583878},\n",
              " 'boolq': {'alias': 'boolq',\n",
              "  'acc,none': 0.6498470948012233,\n",
              "  'acc_stderr,none': 0.008343091327001046},\n",
              " 'lambada_openai': {'alias': 'lambada_openai',\n",
              "  'perplexity,none': 70919.50782999441,\n",
              "  'perplexity_stderr,none': 7092.359624426455,\n",
              "  'acc,none': 0.09819522608189404,\n",
              "  'acc_stderr,none': 0.004145849428034497},\n",
              " 'lambada_standard': {'alias': 'lambada_standard',\n",
              "  'perplexity,none': 352292.3966355288,\n",
              "  'perplexity_stderr,none': 38153.295849679285,\n",
              "  'acc,none': 0.08461090626819329,\n",
              "  'acc_stderr,none': 0.0038772884649187285}}"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ],
      "source": [
        "metrics_pruned"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mhAmm3oH4DPg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "31b71a70-b3c9-4b84-8144-2839c281b683"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:lm_eval.evaluator:pretrained=pretrained=meta-llama/Llama-3.2-1B-Instruct,device=cuda appears to be an instruct or chat variant but chat template is not\n",
            "        applied. Recommend setting `apply_chat_template` (optionally `fewshot_as_multiturn`).\n",
            "WARNING:lm_eval.api.task:[Task: boolq] metric acc is defined, but aggregation is not. using default aggregation=mean\n",
            "WARNING:lm_eval.api.task:[Task: boolq] metric acc is defined, but higher_is_better is not. using default higher_is_better=True\n",
            "WARNING:lm_eval.evaluator:Overwriting default num_fewshot of arc_easy from None to 0\n",
            "WARNING:lm_eval.evaluator:Overwriting default num_fewshot of boolq from None to 0\n",
            "WARNING:lm_eval.evaluator:Overwriting default num_fewshot of lambada_standard from None to 0\n",
            "WARNING:lm_eval.evaluator:Overwriting default num_fewshot of lambada_openai from None to 0\n",
            "100%|██████████| 2376/2376 [00:03<00:00, 641.83it/s]\n",
            "100%|██████████| 3270/3270 [00:01<00:00, 1886.22it/s]\n",
            "100%|██████████| 5153/5153 [00:09<00:00, 529.46it/s]\n",
            "100%|██████████| 5153/5153 [00:09<00:00, 518.70it/s]\n",
            "Running loglikelihood requests: 100%|██████████| 26347/26347 [44:07<00:00,  9.95it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "bootstrapping for stddev: perplexity\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:00<00:00, 94.15it/s]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "bootstrapping for stddev: perplexity\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n",
            "100%|██████████| 1/1 [00:00<00:00, 101.44it/s]\n"
          ]
        }
      ],
      "source": [
        "metrics_base= evaluate_hf_model(\"meta-llama/Llama-3.2-1B-Instruct\", tasks=tasks)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bboB7uU39l_Z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e6d911c8-b0d5-4496-fc30-0dfa364ff46a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'arc_easy': {'alias': 'arc_easy',\n",
              "  'acc,none': 0.6851851851851852,\n",
              "  'acc_stderr,none': 0.009530150430975505,\n",
              "  'acc_norm,none': 0.6313131313131313,\n",
              "  'acc_norm_stderr,none': 0.009899640855681058},\n",
              " 'boolq': {'alias': 'boolq',\n",
              "  'acc,none': 0.6951070336391437,\n",
              "  'acc_stderr,none': 0.008051783411024705},\n",
              " 'lambada_openai': {'alias': 'lambada_openai',\n",
              "  'perplexity,none': 6.578904811488175,\n",
              "  'perplexity_stderr,none': 0.3159444429362019,\n",
              "  'acc,none': 0.5977100718028333,\n",
              "  'acc_stderr,none': 0.006831670941073277},\n",
              " 'lambada_standard': {'alias': 'lambada_standard',\n",
              "  'perplexity,none': 13.07208941627685,\n",
              "  'perplexity_stderr,none': 0.665570718637886,\n",
              "  'acc,none': 0.47952648942363674,\n",
              "  'acc_stderr,none': 0.006960135424338517}}"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ],
      "source": [
        "metrics_base"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "def plot_model_comparison(metrics_base, metrics_pruned):\n",
        "\n",
        "    tasks_to_plot = ['boolq', 'lambada_openai', 'lambada_standard', 'arc_easy']\n",
        "    display_labels = ['BoolQ', 'Lambada OpenAI', 'Lambada Standard', 'arc_easy']\n",
        "\n",
        "    try:\n",
        "        base_scores = [metrics_base[task]['acc,none'] for task in tasks_to_plot]\n",
        "        pruned_scores = [metrics_pruned[task]['acc,none'] for task in tasks_to_plot]\n",
        "    except KeyError as e:\n",
        "        print(f\"Error: Key not found {e}.\")\n",
        "        print(f\"Be sure all tasks ({tasks_to_plot})\")\n",
        "        print(\"and variable 'acc,none' exists in both diccionaries\")\n",
        "        return\n",
        "\n",
        "    n_groups = len(tasks_to_plot)\n",
        "    index = np.arange(n_groups)\n",
        "    bar_width = 0.35\n",
        "\n",
        "    fig, ax = plt.subplots(figsize=(10, 6))\n",
        "\n",
        "    color_base = '#3366CC'\n",
        "    color_pruned = '#DC3912'\n",
        "\n",
        "    ax.bar(index - bar_width / 2,\n",
        "           base_scores,\n",
        "           bar_width,\n",
        "           label='Base Model',\n",
        "           color=color_base)\n",
        "\n",
        "    ax.bar(index + bar_width / 2,\n",
        "           pruned_scores,\n",
        "           bar_width,\n",
        "           label='Pruned Model',\n",
        "           color=color_pruned)\n",
        "\n",
        "    ax.set_title('Base Model vs Pruned Model', fontsize=18, pad=20)\n",
        "\n",
        "    ax.set_ylim([0, 0.9])\n",
        "    ax.set_yticks(np.arange(0, 0.9, 0.2))\n",
        "    ax.tick_params(axis='both', which='major', labelsize=12)\n",
        "\n",
        "    ax.set_xticks(index)\n",
        "    ax.set_xticklabels(display_labels)\n",
        "\n",
        "    ax.yaxis.grid(True, linestyle='-', linewidth=0.5, color='lightgray')\n",
        "    ax.set_axisbelow(True)\n",
        "\n",
        "    ax.legend(fontsize=12, loc='upper center', bbox_to_anchor=(0.5, 1.1),\n",
        "              ncol=2, frameon=False)\n",
        "\n",
        "    plt.tight_layout()\n",
        "    plt.show()"
      ],
      "metadata": {
        "id": "vnANsly5qCx9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plot_model_comparison(metrics_base, metrics_pruned)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 606
        },
        "id": "Wiz0Y5DbqEup",
        "outputId": "9dc79497-c70b-41f5-f056-62b8c71bdb7c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA90AAAJNCAYAAAAs3xZxAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAXAtJREFUeJzt3XlcVNXj//E3jMiggriAQiLumlsUKqW5a+byMSuszFQ00z7VJzPNsl2zzOzjkpVlGu5WYlpprqXmVmlpap/KJRB3yw1cQIHz+8Pf3K/jADLqFbHX8/HgoXPvOfeee+cemPecu/gYY4wAAAAAAMAV55vfDQAAAAAA4HpF6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgD8ozVr1kw+Pj569dVXr+hyX331Vfn4+KhZs2ZXdLkomP6JxwN9CwDOIXQDwCVwfejL7qdIkSKqWrWqevToobVr1+Z3U6+aC/fJJ598ctE67du3d6uTlJRkf0PhtZyOd6fTqXLlyqljx4767LPPZIzJ76Zel+hbAFCwEboB4DKVKVPG+gkJCdGZM2e0Y8cOTZ06VY0aNbriozwFRXx8fK7z9+3bp8WLF1+l1uBKOf949/Hx0d69e/XVV1/p/vvvV/v27ZWenp7fTbzu0bcAoGAhdAPAZTpw4ID1c+jQIaWnp2v16tWKjo6WJA0ZMuQfNeJdunRpFS1aVMuWLdOePXtyLDd16lRlZmaqQoUKV69xuGznH+8nT57U1q1b1bp1a0nSwoUL9eKLL+ZzC69f9C0AKJgI3QBwhTkcDjVq1Ejz5s2zpn3xxRf516CrrGjRooqNjVVWVpYmT56cYznXaF1cXNzVaRiuOF9fX9WqVUtffvmlqlSpIkn68MMPlZGRkc8tuz7RtwCgYCJ0A4BNypUrp1KlSkmSTpw44TH/7Nmz+vLLL9WnTx/Vq1dPYWFhKly4sEJDQ9WmTRvNmjUr12tk9+zZo/79+6tWrVoqWrSo/P39FR4erujoaPXv31/r16/Pse6CBQt077336oYbbpC/v79KlCihJk2aaPz48Tpz5sxlb3vPnj0lKcdgsHr1am3btk2VKlVSkyZNLrq8tLQ0jRkzRg0bNlSJEiXkdDoVGRmp7t27a9OmTbnWzczM1Lhx43TLLbeoaNGiKlmypJo1a6aEhIQ8b8+aNWv00EMPKTIyUk6nU8WLF1eDBg00YsSIbN/bS3X27FmVLl1aPj4+euedd3It+/HHH8vHx0dBQUE6deqUNT0jI0MTJkxQs2bNVLp0afn5+alUqVKqXr267r//fk2aNOmKtdfF6XSqc+fOkqTU1FT9/vvvkqSkpCS3a4p37typPn36qGLFivL393cbia1QoYJ8fHxyDZNxcXHy8fHJNkyeX//MmTMaOXKkbrrpJhUtWlTFixdXixYttGjRootuy+W81wsXLlTr1q0VHBysYsWK6aabbtJbb72ls2fPXnS9eUXfAoACyAAAvPbKK68YSSa3X6N79uyxyowdO9Zj/vLly635kkxQUJAJDAx0m9a5c2eTmZnpUXfTpk2mRIkSVjmHw2FKlChhfHx8rGk9evTwqHfq1CkTGxvrsd7z6916663myJEjl7xPIiMjTVZWlqlcubKRZFauXOlRtlevXkaSGTp0qNt+SExMzHY/1q5d2yrj5+dnihcvbr329fU177zzTrZtSktLM23atHErGxwcbG3vs88+a5o2bWokmVdeecWjfmZmpnnyySfd9lexYsWMw+GwXlevXt0kJSXluD+aNm3q1X58/PHHjSRTr169XMs1a9bMSDJxcXHWtIyMDNO6dWu39hYvXtz4+/u7TfNWXo739957zyqzZs0aY4wxiYmJ1rQZM2aYYsWKGUmmSJEipmjRoiYyMtKqHxkZaSSZ+Pj4HNfRo0ePHI9tV/1x48aZmJgY61hxrVOS8fHxMZMmTcp22ZfzXl+4jySZ4OBgU6hQISPJNGnSxAwePPiSjofzl03fct8fl7IvASA/MNINAFdYZmam1q1bp7vvvluSFBoaqu7du3uUK1KkiPr27aulS5fq+PHjOn78uFJSUnT48GGNHTtWQUFBmj17tt59912PugMGDNDRo0d1yy23aN26dTp79qyOHDmitLQ0bdu2TW+//bZq1arlUa9Pnz5KSEhQpUqVNGPGDGu9p06d0hdffKFKlSrp+++/V69evS5rH5w/Gvnxxx+7zTt58qQ+++wz+fr6XvT018zMTN17773aunWrihcvrunTp+vEiRM6duyYdu7cqQ4dOigrK0v9+vXTwoULPeoPHjxYixcvlo+Pj4YNG6ajR4/q6NGjOnDggP79739rxIgRuY7mvfLKK3rnnXcUGhqq9957T4cPH1ZqaqpOnz6t5cuX6+abb9Yff/yhe+65R1lZWd7upmy5jpUNGzZYI8YXSk5O1sqVK93KS9KsWbO0dOlSOZ1OTZw4UampqTp27JhOnz6tgwcP6vPPP1dsbOwVaeeFzr87dsmSJT3m9+3bV7Vq1dL69et18uRJnThxQkuWLLni7Xj55Ze1Z88ezZs3TydPnrRG3m+99VYZY9SvXz8dP37co97lvNdffvmlhgwZIknq3LmzkpOTdfToUaWkpOi9997T999/r/Hjx1+R7aNvAUABlN+pHwAKovNHtcqUKWP9hISEWCM1QUFBpmvXrjmOjF3M7NmzjSRTuXJlj3kBAQFGklm7dm2el/fdd98ZSSY0NNQkJydnW2b37t2maNGiRpLZuHGjV+09fzTOGGOSk5ONr6+vKVq0qElNTbXKffzxx0aSad26tTHG5Doa98knn1jzFi9e7LHOs2fPWqOatWvXdpu3d+9ea6TxpZdeyrbNXbp0sZZ/4WhcYmKicTgcJiAgwGzatCnb+ikpKaZcuXJGkpk7d262++NSRuOqV69uJJnBgwdnO/+NN94wkkz58uVNVlaWNf3f//63kWT69Onj9Tpzc7GR7uPHj5vw8HAjyZQsWdI6O+P8ke7IyEi34+BCV2qk29/f3/z2228e8w8dOmScTqeRZKZPn+4273Lf65o1a1rvdXZnpnzwwQfWfrjckW5j6FuMdAMoaBjpBoDLdPDgQevnr7/+UmZmpiTp1KlTOn78uA4ePHhJy23fvr0kaefOnTpw4IDbvODgYEnS/v3787w817W8Xbt2VURERLZlypUrp+bNm0vSZT9yKCIiQq1atbJG31xcN3nKy2j6p59+Kkm67bbbdMcdd3jML1SokF555RVJ0tatW7VlyxZrXkJCgjIyMhQQEKCBAwdmu/zcHuc2efJkZWZm6s4779RNN92UbZnAwEB16tRJ0uXvr/N169ZNkjRjxoxsr+ufNm2apHPvpY+PjzXddVxceLzY5dixY/rmm2/UokUL7du3T5LUr18/+fp6frx44oknVKxYMdvbFBsbqxo1anhMDwkJ0W233SZJ2rx5s9u8y3mvN2/erP/973+SpBdffDHbbX/kkUd0ww03XNL2ZIe+BQAFC6EbAC6TMcbt5/Tp09q4caN69Oih+fPnq0mTJm53Mj9famqqRo4cqaZNmyo0NFSFCxe2bjxVpEgRq9yFjwfq0KGDJKlHjx4aMGCAVq5c6XYzreysWbNG0rnwXbZs2Rx/li1bJknatWvXpe4Si+umT67TYHfs2KFVq1apRIkS1gfq3GzYsEGS1KpVqxzLNG/eXA6Hw638+f+vV6+egoKCsq1brVq1HMOQa38tWbIk1/3lCjpXYn+5dOvWTT4+Pm6nkbv89NNP+u233yTJ47KFdu3aycfHR19++aXatm2rWbNmWWH4SnEdnz4+PipRooRatWqln376SZL00EMP6YUXXsi2XqNGja5oO3ISExOT47zw8HBJ0pEjR9ymX8577TrOChUqpMaNG2e7Xl9fXzVr1uyStyk79C0AKDgK5XcDAOB643Q6FRUVpYkTJ+rIkSOaO3eu4uLilJyc7PYBddu2bWrZsqVboC5SpIiCg4Ot0TLXKPnJkyfd1vHWW29px44dWr58uUaNGqVRo0bJ4XAoKipK7du3V58+fTw+8LrCV0pKilJSUi66HRcL8Xlx9913q0SJElqzZo22b99u3XG5S5cucjqdF61/6NAhScp1lNDpdKp06dI6ePCgVT6vdaVzo/t79+71mO7aXydPnvTY/9m5EvvLpXz58mratKlWrFihadOmuQU21yh3/fr1PUZ0b7/9do0YMUIvvviiFi1aZN2tu1y5cmrVqpW6d+9unclwqcqUKWP939/fX6VLl9bNN9+srl275rrs0NDQy1pvXgUGBuY4r1Chcx97Lryb+OW8167jrHTp0vL398+xTrly5S66XG/QtwCg4GCkGwBs9Mgjj0iSjh8/rq+//tptXs+ePbVnzx5VqFBBs2fP1uHDh3Xy5EkdOnRIBw4ccPuweuEpxsHBwfr222+1atUqDRo0SI0aNVKhQoX0008/aejQoapatapmzZrlVsd12vv48eM9Ruez+8nt0U155e/vry5dukiSJk6cqKlTp1rbfq1z7a9nn302T/trxYoVWrFihXx8fLRixQqv1zd58mTr0VrS/41iJyQk6PTp05LOPQ7M9b66TkG/0DPPPKPExESNHj1anTp1UmhoqPbs2aPJkyerRYsW6ty582U9wurAgQPWz65du/TTTz9p4sSJFw3zrhHTa9GlvNf55dVXX5WPj88127fy0gdy2t+uke3ExMRrZn8DwJVA6AYAG0VGRlr/T0xMtP6/e/durV27VtK5O07HxsZ63PE5L9flukY2V69erWPHjumLL75QnTp1dPr0afXq1cvtevKyZctK8u5UTVcQPP8nNDRUzZs3z/aOxtlxhYAxY8Zoz549ql27turVq5enuq7R0QtPrz9fWlqaDh8+7FY+Li7Oug40OTnZo8727dut7cnpDuGXsr+upNjYWAUEBCglJUVffPGFpHOn4x46dEh+fn5W4MpOeHi4nnrqKc2dO1cHDx7U5s2b1bt3b0nnQvyVupP2leQahU5LS8uxTHZ3Hc/Jhceu0+nU3LlzJcn6EsPlct5r1zH3999/5/qM++xGfC9Xbn0rLi4u1y9CSpQoIUl69NFH9fbbb2dbJru+df7/L7ZNOc3P774FAFcbp5cDgI26jz2t6D7nrn+ckyTN6fuTNc81/Yl4SfE/ZVP7/8oMnCVpVvZl3N2gwrfFK/rc/aLU/uU92vDhudOBGzVqpF27dmn+/PkaPny4V9sxdOhQVaxYUcYYHTx4UJMnT1a7du301VdfWdeX56RevXqqU6eOdSMmbx5HVq9ePe3evVvffPONXnvttWzLrFixQhkZGZLOnXLt4uvrq6ysLK1fv14nTpxwu4nXjBkz5HQ6lZaWlmOQa9SokVauXKlly5YpLS0tT6fsXkmuG0nNmjVL06ZN0wMPPGCdWt62bVuVLl06z8uqU6eOPvroI/32229as2aNli5dqieffPKi9XY2LCVJ6iap220l3abl1Y7/Xy/zwWjtzKXc4lBJoSWlqS9o59T/uy688tpzoS8rK8vtuuK8ch27aWlpevPNN5WSkqKvv/5ap06dsu6bcDnvtSvkZmRkaNWqVWrZsqVHmaysLFtGay/WtxwOhzWqfKHg4OCLfrGXU9+qV6+epk2bpg0bNnj0LZc9e/bk+GVZfvctALjaGOkGgH+IPn36SDp3J+KLjXSePHnSbdSubdu2euihh9StWzcNHDhQq1atkp+fn8cp7DkZMWKEBgwYoAEDBuihhx7Kc5sfeOABSdK6deuyfaZzRkaGhg4dKkmqXbu2ateubc1zfZA/e/asx0jezJkzrbvD56RXr14qVKiQ/v77b+suzjk5c+aMTpw4cfEN8pLrFPMlS5Zo+/bt1oh3ds99l6T09PRclxcQECBJ2d5h+1o3ZcqUXM94yInr2O3du7duv/12SdKJEyesfSl5vte5XWd84Xtdt25d3XjjjZKk119/PdtnSn/88ceX1Pa8yK1v+fn55Vjv/C+b/vjjD4/5ufWte++9Vw6HQ6dPn85xlHzKlCk5rvta6FsAcDUVvL+6AFAAHDhwQC+++GJ+N8NN06ZNrdNRH3/8cfXv319//vmnNT89PV3ff/+9Bg0apMjISLcbJ10oODhYAQEB1inB5ztw4IBKlSqlgIAARUdHKyEhQW3bttXbb7+tt99+WyEhIVq6dKluv/12BQcHq23bttmuIz09XVu2bLFuTnXnnXeqQ4cO1gfwxMRE3XvvvVq3bp2kczeXO5+Pj49at24t6dxo5/Dhw5Wamqr169dr+/btVrC6cJTtzz//VOfOnVW/fn3rcVxvvfWWunfvrq1bt1rlkpKS1Lx5cxUuXFgBAQHq1atXjqH3hx9+0J133qnixYurSJEiatq0qXUH59y0bt1aZcuWVUZGhh588EGdPn1aJUqUyPHsgk6dOqlXr15auHChjh07Zk0/cuSIhg0bpm+++UaSLvqFw7UkJSVFo0eP1qOPPupxCcblcF3uERcXp5tuukmPP/64pHPvdY0aNbR161ZVqFBBcXFxysjI0KZNmzR06FBVqVJFjRs3dru5netU/+XLlysqKkphYWFyOp1q3ry5hg4dqieeeMJ6nJuU9+Nh9erVql+/voYNGybp3NMOLnRh38rJ+TdPXL9+vfbv36+qVatKkqZPn66ZM2da1/qvXLlS5cuXt/rW2bNntWDBAqv+DTfcYO2vIUOGqHDhwgoJCVH//v2tS1qWLVum4sWLZ7vNderUsW7Cll3funB/b9q0KcftAoCCgNPLAeAyua5PdDn/lOXoPnfnR5Ny9MEHH8jhcGjixIkaM2aMxowZo2LFisnPz0/Hjx93G6U7//nPx48f199//y1jjA4dOqRx48bpxIkT2Y5a+/n5aejQoTpz5ow++eQTde7cWfPnz7eC3q+//qoOHTqobt26Gjp0qJKSkjR69Gi3ZWRlZaljx45avXq1unXrpgULFmj//v1asGCBgoKCVLx4cStU+vr6avTo0dmG9xkzZqhs2bLKysrS888/r5deesn6osB1Z+/w8HDry4eDBw+qYcOGOnXqlJ588kmVLFlSb731lg4ePKhp06Zp2rRpCggIUEBAgMdjpzZv3qxBgwZ5tOHo0aNq0qSJoqOj9corr8jX11fx8fFq0aKFVq1apQYNGuT4fjkcDj344IMaNWqUdWr1fffdl+Ndsk+fPq34+HjrhlSuu+WfH7hiY2Ot67sLghIlSigrK0v/+c9/lJKSkusIqjdKlfq/0+QzMjI0f/581a1bV5s3b9aePXtUp04d+fj46JNPPtH06dPdTtM+v64kt0eFuU71djqd1s31GjdurNtvv13Dhw/P8/GwZcsW3XHHHQoJCVGzZs30zTff6NixY9Z16d5atGiR6tatK+ncmR41atRQfHy8brvtNqWlpalr167q2bOnnE6n2/HSqVMnJSYmqmPHjkpISNDdd5/7nfbqq69q4sSJOnXqlM6ePavDhw9r3LhxGjNmjKRzX0Ts27dPK1euVGJiYrbb7Dr93dW3XCPz1atXz/F3EQAURIx0A8BlOnjwoNvPqVOnVLZsWbVp0ya/m+ahcOHC+uijj7R27VrFxcWpcuXKyszM1IkTJxQaGqpmzZrp5Zdf1ubNm90eB9SqVSuFhIQoNDRUtWvX1uTJk/Xxxx9bI8nnK1WqlDWSvnr1atWuXVujRo2y5i9dulRnzpzRwoUL9eSTT6pjx44ey5g5c6aWLVumxYsX66OPPtKff/6pUaNGWdeVnzhxQhEREerWrZt++umnHK9PDgkJ0d13361q1aopKipKfn5+OnPmjCIjI/Xuu+96lH/zzTd18OBBff311xo2bJiefvpp7dixQzfccIMCAwNVo0YNORwOK/BXq1ZNzzzzjNauXauff/4520cbbd++Xc2bN9eaNWv09NNP66mnntL333+vG264IU9nQ1x4KnlOp5ZL0rhx4zRixAi1a9dOVatWtZ4bHx4ero4dO2rOnDmaPXt2gTq9vFGjRvrss8/0zjvvXFJ91xdGe/bssUa3HQ6H29kC6enp6ty5s3755Rdt2bJFjz32mHXK+JkzZ1SiRAk1bNjQeq/PH8E9X0REhJo1a6agoCD5+voqLCxMkjR27FgVLlxYUt6Ph5dfflnGGK1atco6Lb5s2bL69ddfL2k/fPnll5LOfaH1ySefqEuXLtYXhv/617906623KiAgwDqTpE2bNtq4caPmzp2r1atXKzIyUk8//bQVhqdOnapTp04pLi5OUVFRcjqdKlasmHXmiOtyFkmaP39+tttcvnx53Xbbbdb+dh2XxYsXd9vfV+sZ7wBgFx9z4XNoAABXTL2+ebn5mb02fBh9yXUnT56snj176r333lO1atUknfuSYfr06Vq2bJk+/fRT3XPPPdnWPXr0qDIzM/Xyyy9r1qxZOnr0qNsyJ06cqJ49e2YbAO+66y7t3LnT4+ZTR48eVbVq1TRs2DC98MILHvVc4uLilJCQoBMnTmju3Lnq3Lmz9uzZo61bt6p169bavn27ChUqpIoVK2rkyJEaOHCgpHMjbMHBwfrhhx/clvfmm29q8ODB2rJli2rXrq02bdpoy5Yt2rt3r9so3MiRIzVo0CAtX75czZo108aNG3XLLbdoypQpateundsyBw8erGnTpunUqVPy9fW19ktiYqIqVKiQ47ZdTd7eNM0Orhupecu1Py8UGRmpDz/80PpSLC4uTlOmTNGuXbtUvnx5t7IVKlRQs2bNPB6f5zq13HV8rlixQs2bN9dbb72lZ555xirnev+/+OILdezYMc/HgzFGgYGBuuuuuzzum9C+fXt9/fXXHo8RvBB9AACuHZxeDgC4qAYNGrg95qtLly66+eab9cQTT6hDhw7WKN78+fM1bNgwbdq0ye365vM/lN9///2aOHGievfureeee04tW7bUPffco9jYWCuAb9++Xb/99luO16jmdr35hdq1a6fAwEB9+umn2rRpk+rXr68qVapYz8M+365duxQTE+Mx3TXquWvXLtWuXVu7du1SlSpVPE57rV69utvr7du3S5J69OiRY/uOHz9uPb4JV57rC6NChQqpTJkyql69uscXPYUKFVK5cuUue10XhnbX++r6wimvx0N6erpOnz5tXXN9vurVq+vrr7/2ql30AQDIX4RuAIDXfH191bx5c40dO1bbt29XrVq1tGrVKnXs2FFNmjTR+++/r7CwMPn5+Sk+Pl4zZ8606gYEBOi7777T8uXLtWDBAi1atEiffvqpWrRooSVLlsjhcCgrK0t16tRxOy39fBEREXluq7+/v+655x5NmTJFf/75p1599dXL3fw8c52KO3LkSEVFRWVbJrvHLeHKufALo+z4+/tne8ZFTtcSZ2ZmyuFweEzPbpoka1Q6r8fDxe5C7y36AADkL0I3AOCSuJ7f67oGdM6cOXI6nVq8eLHbjb5cN/U6n6+vr1q2bKmWLVtq1KhReuONN/TCCy9o+fLlatWqlSpXrqxffvlFLVu2vCI3UXrwwQf18ccfy9fX13oMWXYiIyOzfXzS77//bs13/bt161YZY9zad2HdypUrSzp3Q7NWrVpd9nbg6ipRooTbXeBddu3apUqVKnm9vLweDyEhIQoICLBGic+X3fGZF/QBAMg/BedOKgCAa8bZs2e1ZMkSFS5c2Drt1OFwyMfHx+0uz0lJSZo3b55b3Qvv+i3JGgFzjfDdd9992rt3rz766COPsqdPn871OcrZad68uV577TW9++67HnebP1+7du30448/Wo9Kks49s3zChAmqUKGCatasaZXbt2+fEhISrHKnTp3ShAkT3JYXHR2typUr6+233872WcN//fWXV9uBq6ty5cr6/vvv3Z5ZP3/+fO3evfuSlpfX48HhcKhNmzaaN2+ekpOTrfm//fabFi9efEnrpg8AQP5hpBsAcFELFy60RroOHTqkmTNnavv27Xruueesx1K1b99eo0aN0p133qkHH3xQhw4d0nvvvacqVapo8+bN1rKGDh2q7777Tu3bt7eeB/7++++rXLly1l2au3Xrps8++0yPPvqoli9frkaNGikzM1O///67PvvsMy1evPiipwyfz9fXN093Cn/uuec0a9YstW3b1npk2JQpU5SYmKg5c+ZYpyA/8sgjevfdd9W9e3f99NNPCgsL07Rp01SkSBGP9U6cOFFt27ZVrVq11LNnT91www3au3evli9frqCgIH311Vd53g5cXb1791ZCQoLuvPNO3Xfffdq5c6emT59ujd56y5vjYciQIVq0aJEaN26sxx57TBkZGRo3bpxq1arl1p+8WTd9AADyiQEAIAfx8fFGktuP0+k0UVFRZvz48SYrK8ut/KRJk0zVqlWNv7+/qVGjhomPjzevvPKKOf/PzTfffGPuuusuEx4ebgoXLmzCw8NNly5dzLZt29yWdebMGTNixAhTq1Yt4+/vb0qUKGGio6PNkCFDzPHjx3Ntd48ePUzRokVzLZOYmGgkmZEjR7pN37lzp4mNjTXBwcHG6XSaBg0amPnz53vU37Vrl+nYsaMpUqSIKV26tOnXr59ZtGiRkWSWL1/uVnbjxo3mnnvuMaVKlTL+/v4mMjLS3Hfffeabb77x2NeJiYm5tht549qf69evz7XcxY6V//73v+aGG24w/v7+plGjRmbDhg2madOmpmnTplaZ5cuXG0lm9uzZbnVdx1h8fLzb9LwcD8YYs3LlShMdHW0KFy5sKlWqZD744AOP/nSp23V+++gDAGAvHhkGAAAAAIBNuKYbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsAmhGwAAAAAAmxC6AQAAAACwCaEbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsAmhGwAAAAAAmxC6AQAAAACwCaEbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsAmhGwAAAAAAmxC6AQAAAACwSaH8bsClyMrK0r59+xQYGCgfH5/8bg4AAAAA4B/GGKPU1FSFh4fL1zfn8ewCGbr37duniIiI/G4GAAAAAOAfbvfu3SpXrlyO8wtk6A4MDJR0buOCgoLyuTUAAAAAgH+alJQURUREWPk0JwUydLtOKQ8KCiJ0AwAAAADyzcUueeZGagAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANjE69Cdnp6uZ599VuHh4QoICFBMTIyWLl2ap7rLli1T8+bNVbp0aQUHB6tBgwaaNm2a140GAAAAAKAg8Dp0x8XFadSoUeratavGjh0rh8Ohdu3aafXq1bnW+/LLL3XHHXfozJkzevXVV/X6668rICBA3bt31+jRoy95AwAAAAAAuFb5GGNMXgv/+OOPiomJ0ciRIzVw4EBJUlpammrXrq3Q0FCtXbs2x7p33HGHfv31V/3555/y9/eXJGVkZKhGjRoqWrSofvnllzw3OiUlRcWLF9fx48cVFBSU53oAAAAAAFwJec2lXo10JyQkyOFwqE+fPtY0p9Ophx9+WOvWrdPu3btzbVCJEiWswC1JhQoVUunSpRUQEOBNMwAAAAAAKBC8Ct0bN25UtWrVPFJ8gwYNJEmbNm3KsW6zZs3066+/6qWXXtKOHTu0c+dOvfbaa9qwYYMGDRqU63rT09OVkpLi9gMAAAAAwLWukDeF9+/fr7CwMI/prmn79u3Lse5LL72kxMREvf766xo2bJgkqUiRIpozZ47uuuuuXNc7fPhwDRkyxGN6cnKyAgMDvdkEAAAAAAAuW2pqap7KeRW6T58+7XZ6uIvT6bTm58Tf31/VqlVTbGys7rnnHmVmZmrChAl66KGHtHTpUt1666051h08eLCefvpp63VKSooiIiJUvnx5rukGAAAAAFx1eT0D26vQHRAQoPT0dI/paWlp1vycPPHEE/r+++/1888/y9f33Fnt9913n2rVqqV+/frphx9+yLGuv79/tmEfAAAAAIBrmVfXdIeFhWn//v0e013TwsPDs6135swZTZo0Se3bt7cCtyT5+fmpbdu22rBhg86cOeNNUwAAAAAAuOZ5FbqjoqK0bds2j2F01yh1VFRUtvUOHz6sjIwMZWZmesw7e/assrKysp0HAAAAAEBB5lXojo2Nta7FdklPT1d8fLxiYmIUEREh6dwNzn7//XerTGhoqIKDgzV37ly3Ee0TJ07oq6++Uo0aNXhsGAAAAADguuPVNd0xMTHq3LmzBg8erEOHDqlKlSqaMmWKkpKSNGnSJKtc9+7dtXLlShljJEkOh0MDBw7Uiy++qFtvvVXdu3dXZmamJk2apD179mj69OlXdqsAAAAAALgGeBW6JWnq1Kl66aWXNG3aNB09elR169bV/Pnz1aRJk1zrvfDCC6pYsaLGjh2rIUOGKD09XXXr1lVCQoLuvffeS94AAAAAAACuVT7GNRxdgKSkpKh48eI6fvw4jwwDAAAAAFx1ec2lXl3TDQAAAAAA8o7QDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2ITQDQAAAACATQjdAAAAAADYhNANAAAAAIBNCN0AAAAAANiE0A0AAAAAgE0I3QAAAAAA2KRQfjfgn6Be35/yuwkF2oYPo/O7CQAAAABwSRjpBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGzCjdQAAAAAuOFGwJeHGwHjfIx0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATQjcAAAAAADYhdAMAAAAAYBNCNwAAAAAANiF0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATQjcAAAAAADYhdAMAAAAAYBNCNwAAAAAANiF0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATQjcAAAAAADYhdAMAAAAAYBNCNwAAAAAANiF0AwAAAABgk0L53QDgYnY2LJXfTSjQKq89nN9NAAAAAP6xGOkGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCZeh+709HQ9++yzCg8PV0BAgGJiYrR06dI81//000912223qWjRogoODlbDhg317bffetsMAAAAAACueV6H7ri4OI0aNUpdu3bV2LFj5XA41K5dO61evfqidV999VV16dJFERERGjVqlIYNG6a6detq7969l9R4AAAAAACuZYW8Kfzjjz/qk08+0ciRIzVw4EBJUvfu3VW7dm0NGjRIa9euzbHu999/r6FDh+q///2v+vfvf3mtBgAAAACgAPBqpDshIUEOh0N9+vSxpjmdTj388MNat26ddu/enWPdMWPGqGzZsurXr5+MMTpx4sSltxoAAAAAgALAq9C9ceNGVatWTUFBQW7TGzRoIEnatGlTjnW/+eYb1a9fX++8845CQkIUGBiosLAwvfvuuxddb3p6ulJSUtx+AAAAAAC41nl1evn+/fsVFhbmMd01bd++fdnWO3r0qP7++2+tWbNG3377rV555RWVL19e8fHx+s9//iM/Pz/17ds3x/UOHz5cQ4YM8ZienJyswMBAbzYB+MfZtWtXfjcBAADgH4XPX/8MqampeSrnVeg+ffq0/P39PaY7nU5rfnZcp5IfPnxYn3zyie6//35JUmxsrOrUqaNhw4blGroHDx6sp59+2nqdkpKiiIgIlS9f3mPU/dr0d343AP9gkZGR+d2Ea069vj/ldxMKtA0fRud3EwAAtuPz6+Xg89c/Q17PwPbq9PKAgAClp6d7TE9LS7Pm51RPkvz8/BQbG/t/K/f11f333689e/YoOTk5x/X6+/srKCjI7QcAAAAAgGudV6E7LCxM+/fv95jumhYeHp5tvZIlS8rpdKpUqVJyOBxu80JDQyWdOwUdAAAAAIDriVehOyoqStu2bfMYRv/hhx+s+dmuxNdXUVFR+uuvv3TmzBm3ea7rwENCQrxpCgAAAAAA1zyvQndsbKwyMzM1YcIEa1p6erri4+MVExOjiIgISeducPb777+71b3//vuVmZmpKVOmWNPS0tI0Y8YM1axZM8dRcgAAAAAACiqvbqQWExOjzp07a/DgwTp06JCqVKmiKVOmKCkpSZMmTbLKde/eXStXrpQxxprWt29fTZw4UY8//ri2bdum8uXLa9q0adq1a5e++uqrK7dFAAAAAABcI7wK3ZI0depUvfTSS5o2bZqOHj2qunXrav78+WrSpEmu9QICAvTtt99q0KBB+vjjj3Xy5ElFRUVpwYIFatOmzSVvAAAAAAAA1yqvQ7fT6dTIkSM1cuTIHMusWLEi2+mhoaGaPHmyt6sEAAAAAKBA8uqabgAAAAAAkHeEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhfK7AQAAAABwPdnZsFR+N6HAq7z2cH434YphpBsAAAAAAJsQugEAAAAAsAmhGwAAAAAAm3BNNwAAyFf1+v6U300o8DZ8GJ3fTQAA5ICRbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJl6H7vT0dD377LMKDw9XQECAYmJitHTpUq9X3Lp1a/n4+OiJJ57wui4AAAAAAAWB16E7Li5Oo0aNUteuXTV27Fg5HA61a9dOq1evzvMyPv/8c61bt87bVQMAAAAAUKB4Fbp//PFHffLJJxo+fLhGjhypPn366Ntvv1VkZKQGDRqUp2WkpaVpwIABevbZZy+pwQAAAAAAFBRehe6EhAQ5HA716dPHmuZ0OvXwww9r3bp12r1790WX8dZbbykrK0sDBw70vrUAAAAAABQghbwpvHHjRlWrVk1BQUFu0xs0aCBJ2rRpkyIiInKsn5ycrDfffFMff/yxAgIC8rze9PR0paenW69TUlK8aTYAAAAAAPnCq9C9f/9+hYWFeUx3Tdu3b1+u9QcMGKCbb75ZDzzwgDer1fDhwzVkyBCP6cnJyQoMDPRqWcA/za5du/K7CbjOcEwB1x76JYDrTUH4vZaampqncl6F7tOnT8vf399jutPptObnZPny5ZozZ45++OEHb1YpSRo8eLCefvpp63VKSooiIiJUvnx5j1H3a9Pf+d0A/INFRkbmdxOuQfTJy8ExhSuPPnm56Je48uiXyF8F4fdaXs/A9ip0BwQEuJ3m7ZKWlmbNz05GRoaefPJJdevWTfXr1/dmlZIkf3//bMM+AAAAAADXMq9Cd1hYmPbu3esxff/+/ZKk8PDwbOtNnTpVf/zxhz788EMlJSW5zUtNTVVSUpJCQ0NVpEgRb5oDAAAAAMA1zau7l0dFRWnbtm0ew+iuU8ajoqKyrZecnKyzZ8+qUaNGqlixovUjnQvkFStW1JIlSy6h+QAAAAAAXLu8GumOjY3V22+/rQkTJliP/EpPT1d8fLxiYmKsO5cnJyfr1KlTqlGjhiTpgQceyDaQ33333WrXrp0eeeQRxcTEXOamAAAAAABwbfEqdMfExKhz584aPHiwDh06pCpVqmjKlClKSkrSpEmTrHLdu3fXypUrZYyRJNWoUcMK4BeqWLGiOnXqdOlbAAAAAADANcqr0C2dOx38pZde0rRp03T06FHVrVtX8+fPV5MmTexoHwAAAAAABZbXodvpdGrkyJEaOXJkjmVWrFiRp2W5RsIBAAAAALgeeXUjNQAAAAAAkHeEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGzidehOT0/Xs88+q/DwcAUEBCgmJkZLly69aL3PP/9c999/vypVqqQiRYqoevXqGjBggI4dO3Yp7QYAAAAA4JrndeiOi4vTqFGj1LVrV40dO1YOh0Pt2rXT6tWrc63Xp08f/fbbb3rooYf0zjvv6M4779S7776r2267TadPn77kDQAAAAAA4FpVyJvCP/74oz755BONHDlSAwcOlCR1795dtWvX1qBBg7R27doc6yYkJKhZs2Zu06Kjo9WjRw/NmDFDvXv39r71AAAAAABcw7wa6U5ISJDD4VCfPn2saU6nUw8//LDWrVun3bt351j3wsAtSXfffbck6bfffvOmGQAAAAAAFAhejXRv3LhR1apVU1BQkNv0Bg0aSJI2bdqkiIiIPC/vwIEDkqTSpUvnWi49PV3p6enW65SUlDyvAwAAAACA/OJV6N6/f7/CwsI8prum7du3z6uVjxgxQg6HQ7GxsbmWGz58uIYMGeIxPTk5WYGBgV6tE/in2bVrV343AdcZjing2kO/BHC9KQi/11JTU/NUzqvQffr0afn7+3tMdzqd1vy8mjlzpiZNmqRBgwapatWquZYdPHiwnn76aet1SkqKIiIiVL58eY9R92vT3/ndAPyDRUZG5ncTrkH0ycvBMYUrjz55ueiXuPLol8hfBeH3Wl7PwPYqdAcEBLid5u2SlpZmzc+LVatW6eGHH1abNm30+uuvX7S8v79/tmEfAAAAAIBrmVc3UgsLC9P+/fs9prumhYeHX3QZv/zyizp27KjatWsrISFBhQp5lfsBAAAAACgwvArdUVFR2rZtm8cw+g8//GDNz83OnTt15513KjQ0VF9//bWKFSvmXWsBAAAAAChAvArdsbGxyszM1IQJE6xp6enpio+PV0xMjHXn8uTkZP3+++9udQ8cOKA77rhDvr6+Wrx4sUJCQq5A8wEAAAAAuHZ5dW53TEyMOnfurMGDB+vQoUOqUqWKpkyZoqSkJE2aNMkq1717d61cuVLGGGvanXfeqT///FODBg3S6tWrtXr1amtemTJl1Lp16yuwOQAAAAAAXDu8vqB66tSpeumllzRt2jQdPXpUdevW1fz589WkSZNc6/3yyy+SpLfeestjXtOmTQndAAAAAIDrjteh2+l0auTIkRo5cmSOZVasWOEx7fxRbwAAAAAA/gm8uqYbAAAAAADkHaEbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsAmhGwAAAAAAmxC6AQAAAACwCaEbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsAmhGwAAAAAAmxC6AQAAAACwCaEbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsAmhGwAAAAAAmxC6AQAAAACwCaEbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsAmhGwAAAAAAmxC6AQAAAACwCaEbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsAmhGwAAAAAAmxC6AQAAAACwCaEbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsAmhGwAAAAAAmxC6AQAAAACwCaEbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsAmhGwAAAAAAmxC6AQAAAACwCaEbAAAAAACbELoBAAAAALAJoRsAAAAAAJsQugEAAAAAsEmh/G4AAKBg2dmwVH43ocCrvPZwfjcBAABcJYx0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATQjcAAAAAADYhdAMAAAAAYBNCNwAAAAAANiF0AwAAAABgE57TDQAAUMDtbFgqv5tQoFVeezi/mwDgOsZINwAAAAAANiF0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATQjcAAAAAADYhdAMAAAAAYBNCNwAAAAAANiF0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATQjcAAAAAADYhdAMAAAAAYBNCNwAAAAAANiF0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATQjcAAAAAADYhdAMAAAAAYBNCNwAAAAAANiF0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATQjcAAAAAADYhdAMAAAAAYBNCNwAAAAAANiF0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATQjcAAAAAADYhdAMAAAAAYBNCNwAAAAAANiF0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATQjcAAAAAADYhdAMAAAAAYBNCNwAAAAAANiF0AwAAAABgE0I3AAAAAAA2IXQDAAAAAGATr0N3enq6nn32WYWHhysgIEAxMTFaunRpnuru3btX9913n4KDgxUUFKS77rpLf/75p9eNBgAAAACgIPA6dMfFxWnUqFHq2rWrxo4dK4fDoXbt2mn16tW51jtx4oSaN2+ulStX6vnnn9eQIUO0ceNGNW3aVIcPH77kDQAAAAAA4FpVyJvCP/74oz755BONHDlSAwcOlCR1795dtWvX1qBBg7R27doc677//vvavn27fvzxR9WvX1+S1LZtW9WuXVv//e9/9cYbb1zGZgAAAAAAcO3xaqQ7ISFBDodDffr0saY5nU49/PDDWrdunXbv3p1r3fr161uBW5Jq1Kihli1b6rPPPruEpgMAAAAAcG3zaqR748aNqlatmoKCgtymN2jQQJK0adMmRUREeNTLysrS5s2b1atXL495DRo00JIlS5SamqrAwMBs15uenq709HTr9fHjxyVJKSkp3jQ/32SeOZHfTSjQUjNMfjehQCso/eRqok9eHvrk5aNfuqNPXj765eWhT3qiX14e+uTlKwj90tVGY3J/v70K3fv371dYWJjHdNe0ffv2ZVvvyJEjSk9Pv2jd6tWrZ1t/+PDhGjJkiMf07AI+rj8353cDCrrixfO7BbjO0CevAPolrjD65WWiT+IKo09eAQWoX6ampqp4Lu31KnSfPn1a/v7+HtOdTqc1P6d6ki6priQNHjxYTz/9tPU6KytLR44cUalSpeTj45P3DUCBk5KSooiICO3evdvjDAsAVx99Erj20C+Bawt98p/DGKPU1FSFh4fnWs6r0B0QEOB2mrdLWlqaNT+nepIuqa50LqxfGNiDg4Pz1GZcH4KCgvilBVxD6JPAtYd+CVxb6JP/DLmNcLt4dSO1sLAw7d+/32O6a1pOCb9kyZLy9/e/pLoAAAAAABRUXoXuqKgobdu2zeOi9h9++MGan+1KfH1Vp04dbdiwwWPeDz/8oEqVKuV4EzUAAAAAAAoqr0J3bGysMjMzNWHCBGtaenq64uPjFRMTY93YLDk5Wb///rtH3fXr17sF7z/++EPffvutOnfufDnbgOuYv7+/XnnllWzvBwDg6qNPAtce+iVwbaFP4kI+5mL3N7/Afffdp7lz56p///6qUqWKpkyZoh9//FHffPONmjRpIklq1qyZVq5c6Xbr9NTUVN18881KTU3VwIED5efnp1GjRikzM1ObNm1SSEjIld0yAAAAAADymVc3UpOkqVOn6qWXXtK0adN09OhR1a1bV/Pnz7cCd04CAwO1YsUK9e/fX8OGDVNWVpaaNWum0aNHE7gBAAAAANclr0e6AQAAAABA3nh1TTcAAAAAAMg7QjcAAAAAADYhdKPAa9asmZo1a5bfzQBsV6FCBXXo0MH29axYsUI+Pj5asWKF7esCrnX0u6vjav4tf/XVV+Xj43NV1gUAEqEbV8jkyZPl4+Pj9hMaGqrmzZtr4cKF+dKmw4cP65lnnlH16tXldDpVsmRJtWnTRgsWLMiX9qDgcR3X5z/qEJcuOTlZjz76qCpUqCB/f3+FhoaqU6dOWrNmTX43LUfHjh2T0+mUj4+Pfvvtt2zLxMXFqVixYle5Zdcv+t2Vk5SUpJ49e6py5cpyOp0qW7asmjRpoldeecWt3Pvvv6/JkyfnTyMB4B/A67uXA7kZOnSoKlasKGOMDh48qMmTJ6tdu3b66quvrspIgcsff/yhli1b6q+//lLPnj1Vr149HTt2TDNmzFCHDh307LPP6s0337xq7QH+6dasWaN27dpJknr37q2aNWvqwIEDmjx5sho3bqyxY8fqP//5Tz630tPs2bPl4+OjsmXLasaMGRo2bFh+NwnIkx07dqh+/foKCAhQr169VKFCBe3fv18///yzRowYoSFDhlhl33//fZUuXVpxcXH512AAuI4RunFFtW3bVvXq1bNeP/zwwypTpoxmzZp11UL32bNnFRsbq6NHj+q7775TTEyMNa9///7q2rWrRowYoejoaHXu3PmqtAn4Jzt69KhiY2MVEBCgNWvWqHLlyta8p59+Wm3atNFTTz2l6OhoNWzYMB9b6mn69Olq166dIiMjNXPmTEI3CozRo0frxIkT2rRpkyIjI93mHTp0KJ9adXVkZGQoKytLhQsXzu+moAA5efKkihYtmt/NwHWK08thq+DgYAUEBKhQof/7fufkyZMaMGCAIiIi5O/vr+rVq+vtt9/WhU+vy8jI0GuvvabKlSvL399fFSpU0PPPP6/09PRc1zlnzhxt3bpVzz33nFvgliSHw6EPP/xQwcHBHqfXAZfizJkzevnllxUdHa3ixYuraNGiaty4sZYvX+5WLikpST4+Pnr77bf13nvvqVKlSipSpIjuuOMO7d69W8YYvfbaaypXrpwCAgJ011136ciRI9muc8mSJYqKipLT6VTNmjX1+eefu80/cuSIBg4cqDp16qhYsWIKCgpS27Zt9csvv3gsa8+ePerUqZOKFi2q0NBQ9e/fP9s+tmrVKnXu3Fnly5eXv7+/IiIi1L9/f50+ffqi++jDDz/UgQMHNHLkSLfALUkBAQGaMmWKfHx8NHToUGu66xTj7777Tn379lWpUqUUFBSk7t276+jRox7rWLhwoRo3bqyiRYsqMDBQ7du316+//upWxnUa+N69e9WpUycVK1ZMISEhGjhwoDIzMz2WmZycrFWrVumBBx7QAw88oMTERK1du/ai2wv70e8u3u927typcuXKeQRuSQoNDbX+X6FCBf36669auXKldXmY69rqvG6T63r0zz77TK+//rrKlSsnp9Opli1baseOHR7rnzBhgipXrqyAgAA1aNBAq1at8ihzKe/xmDFjrM8M//vf/yRJq1evVv369eV0OlW5cmV9+OGHF913KLh27dqlxx57TNWrV1dAQIBKlSqlzp07Kykpya2c62/MypUr9dhjjyk0NFTlypWz5i9cuFBNmzZVYGCggoKCVL9+fc2cOdOrthw7dkxPPfWU9Xm3SpUqGjFihLKystzKvf3222rYsKFKlSqlgIAARUdHKyEhwWN5S5cu1e23367g4GAVK1ZM1atX1/PPPy9JOnHihIoWLap+/fp51NuzZ48cDoeGDx/uVftxhRngCoiPjzeSzLJly8xff/1lDh06ZLZu3Wr69u1rfH19zZIlS4wxxmRlZZkWLVoYHx8f07t3b/Puu++af/3rX0aSeeqpp9yW2aNHDyPJxMbGmvfee890797dSDKdOnVyK9e0aVPTtGlT6/WDDz5oJJmkpKQc2+ta9o4dO67cTsB1x3Vcr1+/Pscyf/31lwkLCzNPP/20GT9+vHnrrbdM9erVjZ+fn9m4caNVLjEx0UgyUVFRpmbNmmbUqFHmxRdfNIULFza33nqref75503Dhg3NO++8Y5588knj4+Njevbs6bauyMhIU61aNRMcHGyee+45M2rUKFOnTh23PmaMMevXrzeVK1c2zz33nPnwww/N0KFDzQ033GCKFy9u9u7da5U7deqUqVatmnE6nWbQoEFmzJgxJjo62tStW9dIMsuXL7fK/uc//zHt2rUzb7zxhvnwww/Nww8/bBwOh4mNjb3ofmzYsKFxOp0mLS0txzJNmzY1fn5+5tSpU277vk6dOqZx48bmnXfeMY8//rjx9fU1TZo0MVlZWVbdqVOnGh8fH3PnnXeacePGmREjRpgKFSqY4OBgk5iYaJXr0aOHcTqdplatWqZXr15m/Pjx5t577zWSzPvvv+/RpjfffNMUK1bMalPlypXNY4895lGuR48epmjRohfdD8gb+t1yq+zl9Ls+ffoYh8Nhvvnmm1zLzZ0715QrV87UqFHDTJs2zUybNs3arrxu0/Lly40kc/PNN5vo6GgzevRo8+qrr5oiRYqYBg0auK1v4sSJRpK135966ikTHBxsKlWq5Pa33Nv3uGbNmqZSpUrmzTffNKNHjza7du0ymzdvNgEBAaZ8+fJm+PDh5rXXXjNlypSx9jWuP7NnzzY33XSTefnll82ECRPM888/b0qUKGEiIyPNyZMnrXKu3zM1a9Y0TZs2NePGjTNvvvmmNc/Hx8fUrl3bvP766+a9994zvXv3Nt26dctzO06ePGnq1q1rSpUqZZ5//nnzwQcfmO7duxsfHx/Tr18/t7LlypUzjz32mHn33XfNqFGjTIMGDYwkM3/+fKvM1q1bTeHChU29evXM2LFjzQcffGAGDhxomjRpYpXp2rWrKVOmjMnIyHBb/ltvvWV8fHzMrl27vNmVuML4jYMrwvXL68Iff39/M3nyZKvcvHnzjCQzbNgwt/qxsbHGx8fHCsGbNm0ykkzv3r3dyg0cONBIMt9++6017cLQHRUVZYoXL55re0eNGmUkmS+//PIStxj/BHn58J+RkWHS09Pdph09etSUKVPG9OrVy5rm+mAYEhJijh07Zk0fPHiwkWRuuukmc/bsWWt6ly5dTOHChd2CamRkpJFk5syZY007fvy4CQsLMzfffLM1LS0tzWRmZrq1KTEx0fj7+5uhQ4da08aMGWMkmc8++8yadvLkSVOlShWPD/+u4Hm+4cOH5+kPeXBwsLnppptyLfPkk08aSWbz5s3GmP/b99HR0ebMmTNWubfeestIMl988YUxxpjU1FQTHBxsHnnkEbflHThwwBQvXtxtuuvLtvP3gTHGCgoXqlOnjunatav1+vnnnzelS5d2e59cyyV0Xzn0u+XW9Mvpd1u3bjUBAQHWlw79+vUz8+bNcwseLrVq1XL7O+rtNrlC94033uj2vowdO9ZIMlu2bDHGGHPmzBkTGhpqoqKi3MpNmDDBSHJrg7fvcVBQkDl06JBb+U6dOhmn0+m2r/73v/8Zh8NB6L5OZddn1q1bZySZqVOnWtNcv2duv/12t5B67NgxExgYaGJiYszp06fdlnP+l70X89prr5miRYuabdu2uU1/7rnnjMPhMMnJyTm2+cyZM6Z27dqmRYsW1rTRo0cbSeavv/7KcZ2LFy82kszChQvdptetWzfb/o2ri9PLcUW99957Wrp0qZYuXarp06erefPm6t27t3Ua3tdffy2Hw6Enn3zSrd6AAQNkjLHudP71119LOne954XlJOV6B/LU1FQFBgbm2k7X/NTUVC+2DvDkcDis6wazsrJ05MgRZWRkqF69evr55589ynfu3FnFixe3XrsugXjooYfcLsOIiYnRmTNntHfvXrf64eHhuvvuu63XrlOuN27cqAMHDkiS/P395et77td7ZmamDh8+bJ2Kdn6bvv76a4WFhSk2NtaaVqRIEfXp08ej3QEBAdb/T548qb///lsNGzaUMUYbN27MdR950ydTUlLcpvfp00d+fn7W63//+98qVKiQ9Tti6dKlOnbsmLp06aK///7b+nE4HIqJifE4FVWSHn30UbfXjRs31p9//uk2bfPmzdqyZYu6dOliTXOtY/HixbluC+xHv7t4v6tVq5Y2bdqkhx56SElJSRo7dqw6deqkMmXK6KOPPsq1rktet8mlZ8+ebtdRN27cWJKs/rVhwwYdOnRIjz76qFu5uLg4t/dH8v49vvfeexUSEmK9zszM1OLFi9WpUyeVL1/emn7jjTeqTZs2edp+FDzn95mzZ8/q8OHDqlKlioKDg7M9bh555BE5HA7r9dKlS5WamqrnnntOTqfTraw3j5mbPXu2GjdurBIlSrj9bWrVqpUyMzP13XffZdvmo0eP6vjx42rcuLFbe4ODgyVJX3zxhcfp6S6tWrVSeHi4ZsyYYU3bunWrNm/erIceeijPbYc9CN24oho0aKBWrVqpVatW6tq1qxYsWKCaNWvqiSee0JkzZ7Rr1y6Fh4d7fAC/8cYbJZ27Fsf1r6+vr6pUqeJWrmzZsgoODrbKZScwMPCiYdo1//zr2oBLNWXKFNWtW1dOp1OlSpVSSEiIFixYoOPHj3uUPf/DnyTrg2ZERES20y+8frlKlSoef/irVasmSdY1a1lZWRo9erSqVq0qf39/lS5dWiEhIdq8ebNbm3bt2pXt8qpXr+7R7uTkZMXFxalkyZLWtdBNmzaVpGy383ze9MkLfzdUrVrV7XWxYsUUFhZmbev27dslSS1atFBISIjbz5IlSzxuGOV0Ot0+mEtSiRIlPPbz9OnTVbRoUVWqVEk7duzQjh075HQ6VaFCBbcPNMg/9Lvc+52rjdOmTdPff/+tzZs364033lChQoXUp08fLVu27KL187pNLhfu5xIlSkj6v/3p+tt9Yb/28/NTpUqVPJbnzXtcsWJFt9d//fWXTp8+7bEuKft9jevD6dOn9fLLL1vXUbuO2WPHjuXpuNm5c6ckqXbt2pfVju3bt2vRokUef5datWolyf1mhvPnz9ett95qPd42JCRE48ePd2vv/fffr0aNGql3794qU6aMHnjgAX322WduAdzX11ddu3bVvHnzdOrUKUnSjBkz5HQ6uXHwNYC7l8NWvr6+at68ucaOHWt9OPaGN98qutSsWVObNm1ScnKyxwcAl82bN0tStn/kAW9Mnz5dcXFx6tSpk5555hmFhoZaNyxx/fE+3/nfqOdlurngBoN58cYbb+ill15Sr1699Nprr6lkyZLy9fXVU089leM35LnJzMxU69atdeTIET377LOqUaOGihYtqr179youLu6iy7zxxhu1ceNGpaeny9/fP9symzdvlp+fX7YfkHPjWve0adNUtmxZj/nnj2JKOe/n8xljNGvWLJ08eVI1a9b0mH/o0CGdOHGCZ3PnI/rdxfvd+RwOh+rUqaM6derotttuU/PmzTVjxgwrAFypbbqS+9Pb9/j80UL8c/3nP/9RfHy8nnrqKd12220qXry4fHx89MADD2R7zNp13GRlZal169YaNGhQtvNdX9qtWrVKHTt2VJMmTfT+++8rLCxMfn5+io+Pd7txW0BAgL777jstX75cCxYs0KJFi/Tpp5+qRYsWWrJkidX3unfvrpEjR2revHnq0qWLZs6cqQ4dOnicSYKrj9AN22VkZEg6d2fFyMhILVu2zON0099//12SrLusRkZGKisrS9u3b7dGwSXp4MGDOnbsWLZ3Y3X517/+pZkzZ2rq1Kl68cUXPeanpKToiy++0C233ELoxmVLSEhQpUqV9Pnnn7t9SWTX3fF37NghY4zburZt2ybp3F2IXW1q3ry5Jk2a5Fb32LFjKl26tPU6MjJSW7du9VjeH3/84VZvy5Yt2rZtm6ZMmaLu3btb05cuXZqnNnfo0EHr1q3T7Nmzsz3FLSkpSatWrVKrVq08PgBt375dzZs3t16fOHFC+/fvt5757bobemho6EUDRF6tXLlSe/bs0dChQ91+/0jnRuz69OmjefPmcbpePqLfXTrXYz33799vTcvpC+68blNeuf52b9++XS1atLCmnz17VomJibrpppvc1n0573FISIgCAgKy/cL/wn2N60dCQoJ69Oih//73v9a0tLQ0HTt2LE/1XX9Ttm7d6nG2pTcqV66sEydOXPTv0pw5c+R0OrV48WK3L6Xj4+M9yvr6+qply5Zq2bKlRo0apTfeeEMvvPCCli9fbq2ndu3auvnmmzVjxgyVK1dOycnJGjdu3CVvB64cTi+Hrc6ePaslS5aocOHCuvHGG9WuXTtlZmbq3XffdSs3evRo+fj4qG3btpJkfaAeM2aMW7lRo0ZJktq3b5/jOu+9917VqlVLb775pjZs2OA2LysrS//+97919OhRvfDCC5e7eYD17fL5Izk//PCD1q1bZ8v69u3bp7lz51qvU1JSNHXqVEVFRVkjvQ6Hw2Nkafbs2R7XqbZr10779u1zezTJqVOnNGHCBLdy2W2jMUZjx47NU5v79u2r0NBQPfPMMx7XTqelpalnz54yxujll1/2qDthwgSdPXvWej1+/HhlZGRYvyvatGmjoKAgvfHGG27lXP766688tfF8rlPLn3nmGcXGxrr9PPLII6patSqnmOcz+t3FrVq1Kts+4bofwvmnWBctWjTbUJLXbcqrevXqKSQkRB988IHOnDljTZ88ebLH+i/3PXY4HGrTpo3mzZun5ORka/pvv/3GfRmuY9kds+PGjcv2sZDZueOOOxQYGKjhw4crLS3NbZ43Z2zcd999WrduXbbH2rFjx6wBKYfDIR8fH7f2JSUlad68eW51snuUYVRUlCR5PG6wW7duWrJkicaMGaNSpUpZfy+RvxjpxhW1cOFCa9T60KFDmjlzprZv367nnntOQUFB+te//qXmzZvrhRdeUFJSkm666SYtWbJEX3zxhZ566inrG8abbrpJPXr00IQJE3Ts2DE1bdpUP/74o6ZMmaJOnTq5jXxdyM/PT3PmzFGLFi10++23q2fPnqpXr56OHTummTNn6ueff9bzzz+ve+6556rsExR8H3/8sRYtWuQxvV+/furQoYM+//xz3X333Wrfvr0SExP1wQcfqGbNmjpx4sQVb0u1atX08MMPa/369SpTpow+/vhjHTx40O1b8Q4dOmjo0KHq2bOnGjZsqC1btmjGjBkeZ3Y88sgjevfdd9W9e3f99NNPCgsL07Rp01SkSBG3cjVq1FDlypU1cOBA7d27V0FBQZozZ062z8vOTqlSpZSQkKD27dvrlltuUe/evVWzZk0dOHBAkydP1o4dOzR27Fg1bNjQo+6ZM2fUsmVL3Xffffrjjz/0/vvv6/bbb1fHjh0lnbuh1fjx49WtWzfdcssteuCBBxQSEqLk5GQtWLBAjRo18viSLzfp6emaM2eOWrdu7XETHZeOHTtq7NixOnToEPeFsBH97vL63YgRI/TTTz/pnnvuUd26dSVJP//8s6ZOnaqSJUvqqaeesspGR0dr/PjxGjZsmKpUqaLQ0FC1aNEiz9uUV35+fho2bJj69u2rFi1a6P7771diYqLi4+M9lnkl3uMhQ4Zo0aJFaty4sR577DFlZGRo3LhxqlWrlnWZGa4vHTp00LRp01S8eHHVrFlT69at07Jly1SqVKk81Q8KCtLo0aPVu3dv1a9fXw8++KBKlCihX375RadOndKUKVPytJxnnnlGX375pTp06KC4uDhFR0fr5MmT2rJlixISEpSUlKTSpUurffv2GjVqlO688049+OCDOnTokN577z1VqVLF7RgdOnSovvvuO7Vv316RkZE6dOiQ3n//fZUrV063336727offPBBDRo0SHPnztW///1vt5uRIh9dvRul43qW3SPDnE6niYqKMuPHj3d7zEJqaqrp37+/CQ8PN35+fqZq1apm5MiRHo9iOHv2rBkyZIipWLGi8fPzMxEREWbw4MEez/q98JFhLn/99ZcZMGCAqVKliilcuLDVrkmTJtmyD3D9yelReK6f3bt3m6ysLPPGG2+YyMhI4+/vb26++WYzf/5806NHDxMZGWkty/VYm5EjR7qtw/WondmzZ2e77vMfmxQZGWnat29vFi9ebOrWrWv8/f1NjRo1POqmpaWZAQMGmLCwMBMQEGAaNWpk1q1bl21f2bVrl+nYsaMpUqSIKV26tOnXr59ZtGiRx6OL/ve//5lWrVqZYsWKmdKlS5tHHnnE/PLLL0aSiY+Pz9P+TExMNI888ogpX7688fPzM6VLlzYdO3Y0q1atynHfr1y50vTp08eUKFHCFCtWzHTt2tUcPnzYo/zy5ctNmzZtTPHixY3T6TSVK1c2cXFxZsOGDVaZnB7t9corr1iPD5ozZ85Ff0+sWLHCSDJjx47Ndbm4NPS75Va5y+l3a9asMY8//ripXbu2KV68uPHz8zPly5c3cXFxZufOnW5lDxw4YNq3b28CAwPdHt2V123KaX+69v+FbX3//fdNxYoVjb+/v6lXr5757rvvPJZ5ue+xy8qVK010dLQpXLiwqVSpkvnggw/c+jyuL0ePHjU9e/Y0pUuXNsWKFTNt2rQxv//+u4mMjDQ9evSwyl3s0YRffvmladiwoQkICDBBQUGmQYMGZtasWV61JTU11QwePNj6HFq6dGnTsGFD8/bbb7s9CnPSpEmmatWq1u+W+Ph4j2P0m2++MXfddZcJDw83hQsXNuHh4aZLly4ejyRzadeunZFk1q5d61WbYR8fYy7h7hZAAbRlyxY1btxYERERWr16NTeVAK5hkydPVs+ePbV+/XrrGlQAAHBxd999t7Zs2aIdO3bkd1Pw/3FNN/4x6tSpoy+++ELbt29Xp06d3K4nAwAAAAq6/fv3a8GCBerWrVt+NwXn4Zpu/KM0bdrU48YYAAAAwMWcPn062+d9n69kyZIqXLjwVWrR/0lMTNSaNWs0ceJE+fn5qW/fvle9DcgZoRsAAAAALuLTTz9Vz549cy2zfPlyNWvW7Oo06DwrV65Uz549Vb58eU2ZMsV6sgKuDVzTDQAAAAAXsX//fv3666+5lomOjlaJEiWuUotQUBC6AQAAAACwCTdSAwAAAADAJoRuAAAAAABsQugGAAAAAMAmhG4AAAAAAGxC6AYAAAAAwCaEbgAAAAAAbELoBgAAAADAJv8PGIRjk8ERozwAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YWNOHoVdpgcP"
      },
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7c1FodzbrwMv"
      },
      "source": [
        "**BoolQ:** The performance drop is minimal (from 0.64 to 0.63). BoolQ is a dataset of boolean (true/false) questions based on reading comprehension. The fact that the model nearly maintains its performance suggests that the information required to answer these questions is less sensitive to layer removal or that this type of task primarily benefits from more basic language information, which remains intact after pruning.\n",
        "\n",
        "**Lambada (OpenAI and Standard):** Here, the performance drop is much more pronounced, alsmost dramatic, a drastic reduction. Lambada focuses on predicting the last word in a text, requiring deep contextual understanding and long-term coherence. This result aligns with the earlier test in the notebook, where the model was asked to complete a prompt, and the generated language was not entirely accurate. This difficulty in producing coherent text within a given context negatively impacted its performance in the evaluated task.\n",
        "\n",
        "**ARC Easy:** This benchmark also shows a significant decline (from 0.65 to 0.44). ARC Easy is a reasoning benchmark focused on general knowledge and common sense. The removal of layers likely impacted the model's ability to relate information and maintain reasoning chains, resulting in a reduced capacity to select the correct answer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5MwoMVHLZ7ad"
      },
      "source": [
        "#Conclusion.\n",
        "\n",
        "In this notebook, we explored how depth pruning works on Llama-3.2 Instruct models.\n",
        "\n",
        "After pruning, the model maintains the ability to work as an instruct model, meaning maintaining a dialog with the user, but experiences significant degradation in tasks that require greater contextual and semantic reasoning (Lambada, ARC Easy), while its performance on BoolQ, a comparatively simpler task, remains almost unchanged. This suggests that pruning disproportionately affects the parts of the model that facilitate complex understanding and long-term coherence. BoolQ, being simpler or less dependent on these traits, remains relatively stable, whereas tasks that evaluate context and global coherence are severely impacted.\n",
        "\n",
        "As indicated in the paper: What Matters in Transformers? What Matters in Transformers? Not All Attention is Needed. https://arxiv.org/abs/2406.15786 the best result is achieved by removing the deepest layers of the model.\n",
        "\n",
        "\n",
        "## Future Work.\n",
        "\n",
        "So far, we have explored two forms of structured pruning:\n",
        "\n",
        "- Width pruning: In this approach, neurons from the MLP layers of two model families, DistilGPT and Llama3, were removed. The process of removing neurons from models with GLU architecture works across all families with this structure, such as QWEN, Gemma, or Microsoft Phi.\n",
        "- Depth pruning: Entire blocks were removed from a Llama3 model. This technique can be adapted to all model families.\n",
        "\n",
        "The common point is that we have used very similar methods to decide which elements to remove from the models, based on the absolute weight of the parameters. The next step will involve making these decisions based on metrics generated while the model is running. This will allow us to create models tailored to specific datasets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lW-biFT6U0zQ"
      },
      "source": [
        "##Authors Note.\n",
        "In addition to creating content like this notebook and offering it under the MIT license, I have also contributed to repositories such as those of Hugging Face and Google Gemini.\n",
        "\n",
        "I am especially proud of my book: <a href=\"https://amzn.to/4eanT1g\"><b>Large Language Models:</b> Apply and Implement Strategies for Large Language Models</a> (Apress).\n",
        "\n",
        "You can find it on both <a href=\"https://amzn.to/4eanT1g\">Amazon</a> and <a href=\"https://link.springer.com/book/10.1007/979-8-8688-0515-8\">Springer</a>, where they often have good deals on the purchase price.\n",
        "\n",
        "If you take a look and end up purchasing it, keep in mind that you can reach out with any questions via the Discussions section of this same repository or on any of my social media channels. I’ll do my best to respond as quickly as possible."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZcE2aGubuYDC"
      },
      "source": [
        "## References.\n",
        "- He, S., Sun, G., Shen, Z., & Li, A. (2024). What matters in transformers? not all attention is needed. arXiv preprint arXiv:2406.15786.\n",
        "\n",
        "- Kim, B. K., Kim, G., Kim, T. H., Castells, T., Choi, S., Shin, J., & Song, H. K. (2024). Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834, 11.\n",
        "\n",
        "- Martra, P. (2024). EXPLORING GLU EXPANSION RATIOS: STRUCTURED PRUNING IN LLAMA-3.2 MODELS. https://doi.org/https://doi.org/10.31219/osf.io/qgxea"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "XOCCcMCkrC5Q"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "L4",
      "provenance": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.13"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}